GenAI/LLM/AI QA Test Engineer
REMOTE Nationwide
12 Months Contract with Possible Extension
Overview:
We are seeking an experienced LLM Quality Assurance Test Engineer to evaluate, validate, and enhance our generative AI platforms specifically for our Experience applications. These role is critical for ensuring our generative AI platforms provide accurate, helpful, and magical experiences across all our Experience touchpoints, maintaining our high standards for service and satisfaction.
Responsibilities:
- Design and execute comprehensive test strategies for Large Language Models
- Develop sophisticated prompting techniques to evaluate LLM responses about Disney Experience offerings
- Create test cases that validate accurate knowledge of parks, resorts, and cruise operations
- Identify potential vulnerabilities and edge cases in guest-facing LLM interactions
- Document and track model behaviors, issues, and improvements
- Collaborate with AI/ML teams to improve model performance and safety
Basic Qualifications:
- 3+ years of experience in Quality Assurance, with at least 1 year focused on AI/ML systems
- Demonstrated expertise in LLM evaluation methodologies and prompt engineering
- Strong understanding of LLM limitations, biases, and potential failure modes
- Excellent analytical and problem-solving skills
- Strong documentation and communication abilities
Technical Skills:
- Experience with LLM testing frameworks and evaluation metrics
- Knowledge of prompt engineering best practices
- Understanding of AI safety and ethical considerations
- Familiarity with version control systems and bug tracking tools
- Basic programming skills for test automation (Python preferred)
Key Competencies:
- Creative problem-solving for edge case discovery
- Attention to detail in identifying subtle model behaviors
- Strong analytical thinking for systematic testing approaches
- Ability to think like both a guest and an adversary
- Excellence in cross-functional collaboration
Compensation:
The estimated pay range for this position is USD $80.00/hr – 88.00/hr and is an Exempt role.
Exact compensation and offers of employment are dependent on circumstances of each case and will be determined based on job-related knowledge, skills, experience, licenses or certifications, and location.
Benefits:
We offer comprehensive benefit options which vary depending on role, location, and employment type. The Talent Acquisition Partner can share more details about compensation or benefits for the specific role during the hiring process.
